Real-Time Billboard Substitution in a Video Stream

We present a system that accepts as input a continuous stream of TV broadcast images from sports events. The system automatically detects a predetermined billboard in the scene, and replaces it with a user defined pattern, with no cooperation from the billboards or camera operators. The replacement is performed seamlessly so that a viewer should not be able to detect the substitution. This allows the targeting of advertising to the appropriate audience, which is especially useful for international events. This requires several modules using state of the art computer graphics, image processing and computer vision technology. The system relies on modular design, and on a pipeline architecture, in which the search and track modules propagate their results in symbolic form throughout the pipeline buffer, and the replacement is performed at the exit of the pipe only, therefore relying on accumulated information. Also, images are processed only once. This allows the system to make replacement decisions based on complete sequences, thus avoiding mid-sequence on screen billboard changes. We present the algorithms and the overall system architecture, and discuss further applications of the technology.